Image processing device, endoscope apparatus, image processing method, and program

ABSTRACT

An image processing device includes a memory configured to store a plurality of captured images; and a processor comprising hardware, wherein the processor is configured to implement: an image selection unit configured to: select a reference image from among the plurality of captured images stored in the memory; and select images to be combined with the reference image from: a first group of images among the plurality of captured images, the first group of images being captured earlier than the reference image; and a second group of images among the plurality of captured images, the second group of images being captured later than the reference image; and a composition processing unit configured to compose the images to be combined selected by the image selection unit with the reference image selected by the image selection unit to generate a composite image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 14/446,829, filed on Jul. 30, 2014, and claimsbenefit of priority Japanese Patent Application No. 2013-172634, filedon Aug. 22, 2013. In this application, the contents of theabove-identified U.S. patent application and the Japanese applicationare incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing device, anendoscope apparatus, an image processing method, and a program.

2. Description of Related Art

A method has been known which sets the image that is captured earliestas a reference image and includes using only images that are capturedlater than the reference image, or which sets the image that is capturedlatest to a reference image and includes using only images that arecaptured earlier than the reference image, among a plurality of imagesstored in a frame memory in a process of composing a plurality of imagesto remove noise in the time direction (for example, see JapaneseUnexamined Patent Application, First Publication No. 2012-186593).

SUMMARY OF THE INVENTION

According to a first aspect of the invention, an image processing deviceincludes a memory configured to store a plurality of captured images;and a processor comprising hardware, wherein the processor is configuredto implement: an image selection unit configured to: select a referenceimage from among the plurality of captured images stored in the memory;and select images to be combined with the reference image from: a firstgroup of images among the plurality of captured images, the first groupof images being captured earlier than the reference image; and a secondgroup of images among the plurality of captured images, the second groupof images being captured later than the reference image; and acomposition processing unit configured to compose the images to becombined selected by the image selection unit with the reference imageselected by the image selection unit to generate a composite image.

According to a second aspect of the invention, an endoscope apparatusincludes an insertion unit; an imaging unit disposed at a tip of theinsertion unit; an image processing device, wherein the image processingdevice includes: a memory configured to store a plurality of capturedimages which are captured by the imaging portion, a processor comprisinghardware, wherein the processor is configured to implement: an imageselection unit configured to: select a reference image from among theplurality of captured images stored in the memory; and select images tobe combined with the reference image from: a first group of images amongthe plurality of captured images, the first group of images beingcaptured earlier than the reference image; and a second group of imagesamong the plurality of captured images, the second group of images beingcaptured later than the reference image; and a composition processingunit configured to compose the images to be combined selected by theimage selection unit with the reference image selected by the imageselection unit to generate a composite image, and an image display unitconfigured to display the composite image.

According to a third aspect of the invention, an image processing methodincludes selecting a reference image from among the plurality ofcaptured images stored in the memory; selecting images to be combinedwith the reference image from: a first group of images among theplurality of captured images, the first group of images being capturedearlier than the reference image; and a second group of images among theplurality of captured images, the second group of images being capturedlater than the reference image; and among the plurality of images; andcomposing the images to be combined with the selected reference image togenerate a composite image.

According to a fourth aspect of the invention, a computer-readabledevice stores a program causing a computer to perform steps of:selecting a reference image from among the plurality of captured imagesstored in the memory; selecting images to be combined with the referenceimage from: a first group of images among the plurality of capturedimages, the first group of images being captured earlier than thereference image; and a second group of images among the plurality ofcaptured images, the second group of images being captured later thanthe reference image; and among the plurality of images; and composingthe images to be combined with the selected reference image to generatea composite image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating the structure of an imageprocessing device according to a first embodiment.

FIG. 2 is a schematic diagram illustrating an example of frames storedin a frame memory in the first embodiment.

FIG. 3 is a schematic diagram illustrating the relationship between thetime when an imaging element captures an image and the position of theimaging element in the first embodiment.

FIG. 4A is a schematic diagram illustrating the procedure of a processaccording to the related art which composes a frame F₀ and a frame F_(N)to generate a composite image.

FIG. 4B is a schematic diagram illustrating the procedure of the processaccording to the related art which composes the frame F₀ and the frameF_(N) to generate the composite image.

FIG. 4C is a schematic diagram illustrating the procedure of the processaccording to the related art which composes the frame F₀ and the frameF_(N) to generate the composite image.

FIG. 5 is a graph illustrating the allowable amount of movement X of theimaging element when the frame F₀ which is captured at a time t₀ is usedas a reference image in the first embodiment.

FIG. 6 is a graph illustrating the allowable amount of movement X of theimaging element when a frame F_(N/2) which is captured at a time t_(N/2)is used as the reference image in the first embodiment.

FIG. 7 is a flowchart illustrating a composite image generation processof the image processing device according to the first embodiment.

FIG. 8 is a block diagram illustrating the structure of an imageprocessing device according to a second embodiment.

FIG. 9 is a flowchart illustrating a composite image generation processof the image processing device according to the second embodiment.

FIG. 10 is a schematic diagram illustrating an example of the movementof an imaging element in the second embodiment.

FIG. 11 is a graph illustrating an example of the relationship between atime t and the amount of movement of the imaging element in the secondembodiment.

FIG. 12 is a schematic diagram illustrating an example of the movementof the imaging element in the second embodiment.

FIG. 13 is a graph illustrating an example of the relationship betweenthe time t and the amount of movement of the imaging element in thesecond embodiment.

FIG. 14 is a block diagram illustrating the structure of an imageprocessing device according to a third embodiment.

FIG. 15 is a flowchart illustrating a composite image generation processof the image processing device according to the third embodiment.

FIG. 16 is a block diagram illustrating the structure of an imageprocessing device according to a fourth embodiment.

FIG. 17 is a flowchart illustrating a composite image generation processof the image processing device according to the fourth embodiment.

FIG. 18 is a graph illustrating an example of the relationship between atime t and the amount of movement of an imaging element in each framestored in a frame memory in the fourth embodiment.

FIG. 19 is a graph illustrating an example of the relationship betweenthe time t and the amount of movement of the imaging element in thefourth embodiment.

FIG. 20 is a schematic diagram illustrating an example of the selectionof a reference image in the fourth embodiment.

FIG. 21 is a block diagram illustrating the structure of an endoscopeapparatus according to a fifth embodiment.

DETAILED DESCRIPTION OF THE INVENTION First Embodiment

Hereinafter, a first embodiment of the invention will be described withreference to the drawings. FIG. 1 is a block diagram illustrating thestructure of an image processing device 10 according to this embodiment.In FIG. 1, the image processing device 10 includes a composition controlunit 11 and a frame memory (memory) 12. The composition control unit 11includes an image selection unit 13 and a composition processing unit 14and generates a composite image. The frame memory 12 stores input videosignals (a plurality of captured images). The video signal is, forexample, a moving image and includes frames which are continuouslycaptured.

When a composition instruction is input, the image selection unit 13selects a reference image, which is a standard for composition, and aplurality of images to be combined (to be composed) with the referenceimage from the frames included in the video signal which is stored inthe frame memory 12. The composition processing unit 14 composes thereference image selected by the image selection unit 13 with the imagesto be combined to generate a composite image. The composite imagegenerated by the composition processing unit 14 is an image obtained byremoving noise in a time direction from the reference image. That is,the composition processing unit 14 composes the images to be combinedwith the reference image to remove noise from the reference image.

Next, an example of the frames stored in the frame memory 12 will bedescribed. FIG. 2 is a schematic diagram illustrating an example of theframes stored in the frame memory 12 in this embodiment. In FIG. 2, theframe memory 12 stores N frames F₀, F₁, F₂, . . . , F_(M−1), F_(M),F_(M+1), . . . , F_(N−2), F_(N−1), F_(N) with different capture times t(N is a natural number and M is an integer that is equal to or greaterthan 0 and equal to or less than N). In addition, in FIG. 2, the framesare captured in the order of the frame F₀ to the frame F_(N), the frameF₀ is captured earliest, and the frame F_(N) is captured latest. Forexample, the frame F₀ is captured at t₀ and the frame F₁ is captured att₁. The other frames F₂ to F_(N) are captured at t₂ to t_(N),respectively.

Next, the video signal input to the frame memory 12 will be described.For example, a video signal which is captured by an imaging elementprovided at the tip of an endoscope apparatus is input to the framememory 12. FIG. 3 is a schematic diagram illustrating the relationshipbetween the time when an imaging element 104 provided at the tip of theendoscope apparatus captures an image and the position of the imagingelement 104. In FIG. 3, the imaging element 104 captures the image of anobject for the period from the time t₀ to the time t_(N). Then, theimaging element 104 outputs the frame F₀ at the time t₀ and outputs theframe F_(N) at the time T_(N) (see FIG. 2). In addition, the position ofthe imaging element 104 is different at the time t₀ and the time t_(N).In FIG. 3, the amount of movement X of the imaging element 104 betweenthe imaging element 104 at the time t₀ and the imaging element 104 atthe time t_(N) is X_(N).

When the object remains stationary, the position of the object in theframe F₀ is different from the position of the object in the frame F_(N)since the imaging element 104 is being moved. Therefore, when the frameF₀ and the frame F_(N) are composed with each other to generate acomposite image, a composition process needs to be performed afterpositioning is performed such that the position of the object in theframe F₀ and the position of the object in the frame F_(N) are alignedwith each other.

FIGS. 4A to 4C are schematic diagrams illustrating the procedure of theknown process of composing the frame F₀ and the frame F_(N) to generatea composite image. In FIGS. 4A to 4C, the frame F₀ output from theimaging element 104 shown in FIG. 3 is the reference image and the frameF_(N) is the images to be combined (see the frames shown in FIG. 2). Inaddition, a letter “A” is captured as the object. The frame F₀ and theframe F_(N) each include noise.

FIG. 4A is a schematic diagram illustrating a superimposed image of theframe F₀ and the frame F_(N). The position of the imaging element 104 isdifferent at the time t₀ when the frame F₀ is captured and at the timet_(N) when the frame F_(N) is captured. Therefore, in FIG. 4A, thedeviation between the position of an object “A” 401 in the frame F₀ andthe position of an object “A” 402 in the frame F_(N) is the amount ofmovement X_(N). The positional deviation between the object “A” in theimages to be combined (frame F_(N)) and the object “A” in the referenceimage (frame F₀) is referred to as the amount of blur.

FIG. 4B is a schematic diagram illustrating two frames, with theposition of the object “A” in the frame F_(N) being aligned with theposition of the object “A” in the frame F₀. In FIG. 4A, the compositionprocessing unit 14 aligns the position of the object “A” 402 in theframe F_(N), which is the images to be combined, with the position ofthe object “A” 401 in the frame F₀ that is the reference image. As such,when the object is positioned in order to move the position of the frameF_(N), a region in which the frame F₀ and the frame F_(N) overlap eachother and a region in which the frame F₀ and the frame F_(N) do notoverlap each other are generated.

FIG. 4C is a schematic diagram illustrating a composite image 500generated by calculating the weighted average of the frame F₀ and theframe F_(N) which are positioned relative to each other. The region ofthe composite image 500 is the same as the region of the frame F₀ whichis the reference image. In FIG. 4C, noise is reduced in the region inwhich the frame F₀ and the frame F_(N) overlap each other. However,since the composite image 500 is generated by calculating the weightedaverage of the frame F₀ and the frame F_(N), the brightness of a region501 in which the frame F₀ and the frame F_(N) do not overlap each otheris low.

Therefore, the maximum amount of blur of the images to be combined isdetermined according to the allowable region 501. For example, when thesize of the allowable region 501 is widened, because the images to becombined with a large amount of blur are used to generate the compositeimage 500, it is possible to use a large number of frames. Accordingly,although it is possible to improve a noise reduction effect, theeffective size of the image without reduced brightness is small. Whenthe allowable region 501 is narrowed, because it is difficult to use theimages to be combined with a large amount of blur to generate thecomposite image 500, the composite image is small. Consequently, thenoise reduction effect is reduced, but the effective size withoutreduced brightness is large. In accordance with these things, themaximum amount of blur is decided. When the amount of movement X_(N) ofthe imaging element 104 is large, the amount of blur of the images to becombined is large. Therefore, the allowable amount of movement X_(N) ofthe imaging element 104 is determined on the basis of the allowablerange of the region 501.

Next, the relationship among the reference image, the allowable amountof movement X per unit time of the imaging element 104, and theallowable range which is selected as the images to be combined will bedescribed.

FIG. 5 is a graph illustrating the allowable amount of movement X perunit time of the imaging element 104 when the oldest frame F₀ which iscaptured at the time t₀ among a plurality of frames stored in the framememory 12 is set as the reference image. In the graph of FIG. 5, thevertical axis indicates the amount of movement per unit time of theimaging element 104 and when the interval between captured frames isconstant, the amount of movement per unit time of the imaging element104 is regarded as the amount of movement per between frames.Hereinafter, the allowable amount of movement X per unit time of theimaging element 104 is simply referred to as the amount of movement X.The horizontal axis of the graph of FIG. 5 indicates a time t. Inaddition, the imaging element 104 moves in a straight line with aconstant speed. The allowable amount of movement X of the imagingelement 104 is referred to as X_(a).

In FIG. 5, the amount of movement X of the imaging element 104 is morethan the amount of movement X_(a) at a time after a past time t_(M).Therefore, the frames F which can be used as the images to be combinedto generate the composite image 500 are the frames F₀ to F_(M) which arecaptured for the period from the time t₀ to the time t_(M) (see FIG. 2).That is, when the frame F₀ is set as the reference image, it isdifficult to use the frames F_(M+1) to F_(N) which are captured at thetimes after the time t_(M) as the images to be combined.

FIG. 6 is a graph illustrating the allowable amount of movement X of theimaging element 104 when the frame F_(N/2) which is captured at a timet_(N/2) is used as the reference image in this embodiment. In the graphof FIG. 6, the vertical axis, the horizontal axis, and the allowableamount of movement X_(a) of the imaging element 104 are the same asthose in the graph of FIG. 5.

In this embodiment, as the reference image, a frame is selected from theframes other than the oldest frame F₀ which is captured at the time t₀and the newest frame F_(N) which is captured at the time t_(N) in theframe memory 12. Specifically, a frame F_(N/2) which is captured at atime t_(N/2) closest to an intermediate (middle) time between the timewhen the frame F₀ is captured and the time when the frame F_(N) iscaptured is selected as the reference image.

In FIG. 6, when the frame F_(N/2) captured at the time t_(N/2) is set tothe reference image, both the position of the imaging element 104 at thetime t_(N/2) when the reference image is captured and the amount ofmovement X based on the position of the imaging element 104 at the timet₀ are not more than the amount of movement X_(a). Both the position ofthe imaging element 104 at the time t_(Nn) when the reference image iscaptured and the amount of movement X based on the position of theimaging element 104 at the time t_(N) are not more than the amount ofmovement X_(a). Therefore, when the composite image 500 is generated,all of the frames before the frame F_(N/2), which is the referenceimage, and all of the frames after the frame F_(N/2), which is thereference image, can be used as the images to be combined. That is, allof the frames F₀ to F_(N) captured from the time t₀ to the time t_(N)can be used as the images to be combined.

As such, since the frame F_(N/2) which is captured at the time closestto the intermediate time t_(N/2) between the capture time t₀ of theoldest frame F₀ and the capture time t_(N) of the newest frame F_(N) isused as the reference image among a plurality of frames that is storedin the frame memory 12, it is possible to use a large number of framesstored in the frame memory 12 as the images to be combined. Therefore, agood noise reduction effect is expected by the composition of a largenumber of frames. In particular, since the images to be combined isselected from a group of the frames which are captured earlier than thereference image and a group of the frames which are captured later thanthe reference image in the frame memory 12, it is possible to use alarge number of frames as the images to be combined.

In this embodiment, the frame F_(N/2) captured at the time closest tothe time t_(N/2) is used as the reference image. However, the inventionis not limited thereto. The reference image may be selected from theframes other than the oldest frame F₀ which is captured at the time t₀and the newest frame F_(N) which is captured at the time t_(N) in theframe memory 12.

Next, a composite image generation process of the image processingdevice 10 will be described. FIG. 7 is a flowchart illustrating thecomposite image generation process of the image processing device 10 inthis embodiment. In this embodiment, the frames are input one by onefrom the outside to the frame memory 12.

(Step S101) The image processing device 10 stores the input frames(video signals) in the frame memory 12. Then, the process proceeds toStep S102.

(Step S102) The composition control unit 11 determines whether acomposition instruction is received. When the composition control unit11 determines that the composition instruction is received, the processproceeds to Step S103. In the other cases, the process returns to StepS101. In the following description, it is assumed that the frame memory12 stores a plurality of frames.

(Step S103) First, the image selection unit 13 selects the referenceimage from a group of the frames other than the frame which is capturedearliest and the frame F_(N) which is captured latest among a pluralityof frames which the image processing device 10 stores in the framememory 12. In this embodiment, the frame which is captured at the timeclosest to the intermediate time between the capture time of the oldestframe and the capture time of newest the frame is selected as thereference image. Then, the process proceeds to Step S104.

(Step S104) The image selection unit 13 selects, as a reference image,one frame which is captured at the time closest to the capture time ofthe frame selected as the reference image from a plurality of framesother than the frame selected as the reference image. The comparativeimage is images to be combined candidate when the images to be combinedare selected. Then, the process proceeds to Step S105. When Step S104 isperformed again after Step S106, the image selection unit 13 selects oneframe which is captured at the time closest to the capture time of theframe selected as the reference image from the frames which have notbeen subjected to the determination process in Step S105. In addition,the image selection unit 13 sequentially selects the reference imagefrom the frame which is captured earlier than the reference image andthe frame which is captured later than the reference image.

(Step S105) The image selection unit 13 determines whether the framewhich is selected as the comparative image in Step S104 can be used asthe images to be combined. The image selection unit 13 sets, as theimages to be combined, the frame which is determined to be used as theimages to be combined. Then, the process proceeds to Step S106. Theimage selection unit 13 determines whether the selected frame can beused as the images to be combined using, for example, the followingmethod. The image selection unit 13 aligns the position of the object inthe reference image and the position of the object in the comparativeimage. The image selection unit 13 calculates the amount of blur of theobject in the comparative image with respect to the object in thereference image. When the calculated amount of blur is less than athreshold value, the image selection unit 13 determines that theselected frame can be used as the images to be combined. The imageselection unit 13 set the frame which is determined to be capable ofbeing used as the images to be combined. On the other hand, when thecalculated amount of blur is equal to or greater than the thresholdvalue, the image selection unit 13 determines that it is difficult touse the selected frame as the images to be combined. The threshold valueis set based on, for example, the allowable range of the region 501shown in FIG. 4C. In addition, the threshold value may be predeterminedor it may be arbitrarily set.

(Step S106) The image selection unit 13 determines whether thedetermination process in Step S105 has been performed for all framesother than the frame which is selected as the reference image among theframes stored in the frame memory 12. When the image selection unit 13determines that the process in Step S105 has been performed for allframes other than the frame selected as the reference image, the processproceeds to Step S107. In the other cases, the process returns to StepS104.

(Step S107) The composition processing unit 14 composes all of theimages to be combined set in Step S105 with the reference image selectedin Step S103 to generate a composite image. Then, the process ends.

As described above, in this embodiment, the image selection unit 13selects, as the reference image, the frame which is captured at the timeclosest to the intermediate time between the capture time of the oldestframe and the capture time of the newest frame among the plurality offrames stored in the frame memory 12. In addition, the image selectionunit 13 can select the frame as the comparative image from a group ofthe frames which are captured earlier than the reference image and agroup of the frames which are captured later than the reference image.Then, the image selection unit 13 sets, as the images to be combined,the frame which can be used as the images to be combined among theframes which are selected as the comparative images. That is, the imageselection unit 13 can select the frame as the images to be combined froma group of the frames which are captured earlier than the referenceimage and a group of the frames which are captured later than thereference image. Therefore, according to this embodiment, the imageprocessing device 10 can generate a composite image, using a largenumber of frames which the image processing device 10 stores in theframe memory 12 as the images to be combined. As a result, a good noisereduction effect is expected.

In this embodiment, as a preferred example in which the reference imageis selected from the frames other than the oldest frame F₀ which iscaptured at the time t₀ and the newest frame F_(N) which is captured atthe time t_(N) in the frame memory 12, the frame which is captured atthe time closest to the intermediate time between the capture time ofthe oldest frame and the capture time of the newest frame among theplurality of frames stored in the frame memory 12 is selected as thereference image. However, the invention is not limited thereto. Forexample, the reference image may be selected such that the number offrames which are captured earlier than the frame set as the referenceimage and are then stored is close to, preferably, equal to the numberof frames which are captured later than the frame set as the referenceimage and are then stored.

In the above-mentioned example, the image selection unit 13 selects allframes (except for the reference image) stored in the frame memory 12,sets the selected frames as the comparative images, and determineswhether all of the comparative images can be used as the images to becombined. However, the invention is not limited thereto. For example,the number of images to be combined may be predetermined. In this case,in Step S106, the number of reference frames which are set as the imagesto be combined is counted and it is determined whether the count resultreaches a predetermined number of images to be combined. Even when thereis a comparative image which has not been subjected to the process ofdetermining whether the comparative image can be used as the images tobe combined, the process may proceed to Step S107 at the time the numberof reference frames which are set as the images to be combined reaches apredetermined value.

As such, when the number of images to be combined is predetermined, thecomparative image is sequentially selected from the frame which iscaptured at a time close to the time when the reference image iscaptured, which is advantageous in improving the efficiency of thecomposition process in the next stage. The comparative image which iscaptured at a time close to the time when the reference image iscaptured is very likely to have a small amount of blur with respect tothe reference image. Therefore, the set images to be combined is verylikely to be preferentially set from the frame which has a small amountof blur with respect to the reference image. As a result, it is possibleto generate a composite image with high efficiency.

Second Embodiment

Next, a second embodiment of the invention will be described withreference to the drawings. In this embodiment, a reference image isselected on the basis of the amount of movement of the object to beestimated by a movement amount estimation unit 21, which will bedescribed below.

FIG. 8 is a block diagram illustrating the structure an image processingdevice 20 according to this embodiment. In FIG. 8, the image processingdevice 20 includes a composition control unit 11, a frame memory 12, andthe movement amount estimation unit 21. The composition control unit 11and the frame memory 12 are the same as those in the first embodiment.The movement amount estimation unit 21 estimates the amount of movementX of the object. There are some methods of estimating the amount ofmovement of the object. For example, the following methods areconsidered: a method which compares the brightness value of a newlyinput frame with the brightness value of a previously input frame toestimate the amount of movement of the object in the newly input frame;and a method which detects the amount of movement X of the imagingelement 104 using a sensor in an environment in which the object hardlymoves and regards the detected amount of movement X of the imagingelement as the amount of movement of the object. In this embodiment, themovement amount estimation unit 21 may use any estimation method.

In this embodiment, frame processing has been described. However, wheninterlaced processing is performed, the amount of movement X of theobject may be estimated using a newly input interlace image and aninterpolated image based on a previously input interlace image.

Next, a composite image generation process according to this embodimentwill be described. FIG. 9 is a flowchart illustrating the compositeimage generation process of the image processing device 20 in thisembodiment. The meaning of the amount of blur is the same as that in thefirst embodiment and the description thereof will not be repeated.

(Step S201) The image processing device 20 stores frames (video signals)which are input one by one from the outside in the frame memory 12.Then, the process proceeds to Step S202.

(Step S202) The movement amount estimation unit 21 estimates the amountof movement X of the object when the frame which the image processingdevice 20 stores in the frame memory 12 in Step S201 is captured. Then,the process proceeds to Step S203. As described above, for example, theamount of movement X of the object can be estimated by comparing theframe which is stored in the frame memory 12 in Step S201 with theprevious frame.

(Step S203) The image processing device 20 stores the amount of movementX which has been estimated by the movement amount estimation unit 21 inStep S202 so as to be associated with the frame stored in Step S201.Then, the process proceeds to Step S204.

(Step S204) The composition control unit 11 determines whether acomposition instruction is received. When the composition control unit11 determines that the composition instruction is received, the processproceeds to Step S205. In the other cases, the process proceeds to StepS201.

(Step S205) The image selection unit 13 selects the reference image froma plurality of frames that are captured when the amount of movement X ofthe object is equal to or less than a prescribed amount of movementX_(a) which is a threshold value. In this embodiment, first, the imageselection unit 13 excludes a frame F₀ which is captured earliest and aframe F_(N) which is captured latest from a plurality of frames that arecaptured when the amount of movement X of the object is equal to or lessthan the prescribed amount of movement X_(a). Next, the image selectionunit 13 selects a frame which is captured when the amount of movement Xof the object is the minimum among the remaining frames, as thereference image. When there are a plurality of frames satisfying theabove-mentioned conditions, a frame which is captured at the timeclosest to an intermediate (middle) time t_(N/2) between the time t₀when the oldest frame F₀ is captured and the time t_(N) when the newestframe F_(N) is captured is used as the reference image. Then, theprocess proceeds to Step S206.

(Step S206) The image selection unit 13 selects one comparative imagefrom a plurality of frames which the image processing device 20 storesin the frame memory 12. Specifically, first, the image selection unit 13excludes the frame which is selected as the reference image and theframe which has been subjected to the process in Step S207 from theframes. Next, the image selection unit 13 selects a frame which iscaptured at a time closest to the time when the reference image iscaptured from the remaining frames. When the estimated amount ofmovement X of the frame which the image selection unit 13 selects isless than the prescribed amount of movement X_(a) which is a thresholdvalue (which will be described below), the image selection unit 13selects the selected frame as the comparative image. That is, the imagesto be combined are selected from a frame group in which the estimatedamount of movement X is less than the prescribed amount of movementX_(a). In other words, the image selection unit 13 reads the estimatedamount of movement X which is stored so as to be associated with eachframe and does not select the frame in which the estimated amount ofmovement X is equal to or more than the prescribed amount of movementX_(a) as the comparative image, that is, the images to be combined. Thisprocess makes it possible to reduce the load of the following process ofdetermining the images to be combined. Then, the process proceeds toStep S207.

(Step S207) The image selection unit 13 determines whether thecomparative image selected in Step S206 can be used as the images to becombined. The process of determining whether the comparative image canbe used as the images to be combined and the process of setting theimages to be combined are the same as those in the first embodiment andthe description thereof will not be repeated. Then, the process proceedsto Step S208.

(Step S208) The image selection unit 13 determines whether the processin Step S207 has been performed for all of the frames other than theframe selected as the standard frame and the frame in which theestimated amount of movement X is equal to or less than the prescribedamount of movement X_(a) among a plurality of frames which the imageprocessing device 20 stores in the frame memory 12. When the imageselection unit 13 determines that the process in Step S207 has beenperformed for all of the frames, the process proceeds to Step S208. Inthe other cases, the process returns to Step S206.

(Step S209) The composition processing unit 14 composes the images to becombined set in Step S207 with the selected reference image to generatea composite image. Then, the process ends.

When the amount of movement X of the object is equal to or more than theprescribed amount of movement X_(a), the amount of blur is greater thanan allowable value in the captured image of the object. Therefore, it ispreferable that the frame which is captured when the amount of movementX of the object is equal to or more than the prescribed amount ofmovement X_(a) is not used as the comparative image in terms of theefficiency of the composition process. That is, it is preferable thatthe image processing device 20 determine whether the comparative imageis used as the images to be combined using only the frame which iscaptured when the amount of movement X of the object is less than theprescribed amount of movement X_(a) in terms of the efficiency of thecomposition process. In other words, the image processing device 20selects the images to be combined from the frames which are capturedwhen the amount of movement X of the object is less than the prescribedamount of movement X_(a), which is preferable in terms of the efficiencyof the composition process.

Therefore, in this embodiment, since the frame which is captured whenthe amount of movement X of the object is equal to or more than theprescribed amount of movement X_(a) is not selected as the comparativeimage, it is possible to reduce the load of the process of determiningthe images to be combined.

The prescribed amount of movement X_(a), which is a threshold value, maybe predetermined or it may be arbitrarily set. For example, when ashutter speed is high, the amount of movement of the object is smalleven though the speed of the imaging element 104 is high. When theshutter speed is low, the amount of movement of the object is likely tobe large even though the speed of the imaging element 104 is low.Therefore, when the shutter speed is high, the prescribed amount ofmovement X_(a) may be set to a smaller value. When the shutter speed islow, the prescribed amount of movement X_(a) may be set to a largervalue. As such, the prescribed amount of movement X_(a) varies dependingon the shutter speed when the frame is captured. Therefore, theprescribed amount of movement X_(a) may vary depending on the shutterspeed.

It is considered that the movement of the object is caused by themovement of the imaging element and the application of this embodimentdepending on the movement of the imaging element 104 will be describedusing the following reference examples in order to promote theunderstanding of this embodiment.

First, Reference example 1 in which the imaging element 104 regularlyvibrates will be described. FIG. 10 is a schematic diagram illustratingan example of the movement of the imaging element 104. FIG. 11 is agraph illustrating the relationship between the capture time t and theamount of movement X_(N) of the imaging element 104 during the captureof each frame when the imaging element 104 vibrates as shown in FIG. 10.In the example shown in the drawings, the imaging element 104 regularlyvibrates from the time t₀ to the time t_(N).

In FIG. 11, the amount of movement X of the imaging element 104 from thetime t₀ to the time t_(N) is constantly less than the prescribed amountof movement X_(a). Therefore, the image selection unit 13 determinesthat all of the frames F₀ to F_(N) captured from the time t₀ to the timet_(N) can be used as the reference image and the comparative image.

Next, a comparative image selection method in Reference example 1 willbe described. As described in Step S205 of FIG. 9, first, the imageselection unit 13 excludes the frame F₀ which is captured earliest andthe frame F_(N) which is captured latest among the frames which arecaptured when the amount of movement X of the imaging element 104 isequal to or less than the prescribed amount of movement X_(a). Then, theimage selection unit 13 selects a frame F_(X) and a frame F_(Z) whichare captured when the amount of movement X of the imaging element 104 isthe minimum from the remaining frames and sets the selected frames asthe candidate frames of the reference image.

As such, when there are a plurality of frames satisfying the comparativeimage selection conditions, a frame which is captured at the timeclosest to an intermediate time t_(N/2) between the time t₀ when theoldest frame F₀ is captured and the time t_(N) when the newest frameF_(N) is captured is used as the reference image. Therefore, inReference example 1, the image selection unit 13 selects the frame F_(Z)as the reference image. A group of the frames other than the referenceimage (frame F_(N)) among the frames F₀ to F_(N) which are captured fromthe time t₀ to the time t_(N) is the candidates of the comparativeimage.

Next, the comparative image selection method will be described. InReference example 1, as shown in FIG. 11, all of the frames, which arethe candidates of the comparative image, are captured when the amount ofmovement is less than the prescribed amount of movement X_(a).Therefore, all of the frames in the frame memory 12 except for thereference image can be selected as the comparative image.

Next, Reference example 2 in which the imaging element 104 movesirregularly will be described. FIG. 12 is a schematic diagramillustrating an example of the movement of the imaging element 104. FIG.13 is a graph illustrating the capture time t and the amount of movementX of the imaging element 104 during the capture of each frame when theimaging element 104 moves irregularly as shown in FIG. 12. In FIG. 12,the imaging element 104 moves irregularly for the period from the timet₀ to the time t_(N).

In FIG. 13, there is a frame which is captured when the amount ofmovement of the imaging element 104 is equal to or more than theprescribed amount of movement X_(a) the frames F₀ to F_(N). For example,a frame F_(Y) is captured when the amount of movement of the imagingelement 104 is equal to or more than the prescribed amount of movementX_(a). As shown in FIG. 13, the amount of blur of an object “A” 403 inthe frame F_(Y) is large.

For example, a frame F_(X) and a frame F_(Z) are captured when theamount of movement X of the imaging element 104 is less than theprescribed amount of movement X_(a). Therefore, the image selection unit13 determines that a group of the frames (including the frame F_(X) andthe frame F_(Z)) which are captured when the amount of movement X of theimaging element 104 is less than the prescribed amount of movement X_(a)can be used as the reference image and the comparative image.

Next, a comparative image selection method in Reference example 2 willbe described. As described in Step S205 of FIG. 9, first, the imageselection unit 13 excludes the frame F₀ which is captured earliest andthe frame F_(N) which is captured latest among the frames which arecaptured when the amount of movement X of the imaging element 104 isless than the prescribed amount of movement X_(a). Then, the imageselection unit 13 selects a frame F_(X) which is captured when theamount of movement X of the imaging element 104 is the minimum from theremaining frames and sets the selected frame as the reference image.

Next, the comparative image selection method will be described. Theimage selection unit 13 sequentially selects the comparative image fromthe frames which are captured earlier than the reference image whiledetermining whether the estimated amount of movement X is less than theprescribed amount of movement X_(a). In Reference example 2, as shown inFIG. 13, a frame in which the estimated amount of movement X is equal toor more than the prescribed amount of movement X_(a) is included in aplurality of frames stored in the frame memory 12. For example, theframe is a frame F_(Y).

The image selection unit 13 reads the estimated amount of movement Xwhich is stored in the frame memory 12 so as to be associated with eachframe, does not select the frame in which the estimated amount ofmovement X is equal to or more than the prescribed amount of movementX_(a), such as the frame F_(Y), as the comparative image, and checks thenext frame or another frame. As a result, the image selection unit 13selects, as the comparative image, only the frame in which the estimatedamount of movement X is less than the prescribed amount of movementX_(a). Therefore, in Reference example 2, the number of frames which areselected as the comparative image is less than that in Referenceexample 1. However, it is obvious that the frame in which the estimatedamount of movement X is equal to or more than the prescribed amount ofmovement X_(a) does not become the images to be combined. As a result,the process of determining whether the frame can be used as the imagesto be combined causes a reduction in the processing speed or an increasein processing load. Therefore, an inadequate frame is not used for theprocess of determining whether the frame can be used as the images to becombined and is not selected as the comparative image in a comparativeimage selection process which is performed before the determinationprocess. As a result, the load of the process of determining whether theframe can be used as the images to be combined and the efficiency of theprocess is improved.

As described above, in this embodiment, the image selection unit 13selects the reference image from a plurality of frames which arecaptured when the amount of movement X of the object is less than theprescribed amount of movement X_(a). Preferably, the frame F₀ which iscaptured earliest and the frame F_(N) which is captured latest areexcluded from a plurality of frames which are captured when the amountof movement X of the object is less than the prescribed amount ofmovement X_(a) and a frame which is captured when the amount of movementX of the object is the minimum among the remaining frames is selected asthe reference image. If there are a plurality of frames which arecaptured when the amount of movement X of the object is the minimum, theimage selection unit 13 selects, as the reference image, a frame whichis captured at the time closest to the intermediate time t_(N/2) betweenthe time t₀ when the oldest frame F₀ is captured and the time t_(N) whenthe newest frame F_(N) is captured.

In addition, the image selection unit 13 selects as the comparativeimage, only the frame which is captured when the amount of movement X ofthe object is less than the prescribed amount of movement X_(a) from theframes which are captured earlier and later with respect to thereference image. That is, the frame which is obviously improper as theimages to be combined and is more than the prescribed amount of movementX_(a) and is more equal to or more than is not used for the process ofdetermining whether the frame can be used as the images to be combined.In other words, the image selection unit 13 does not select the framewhich is obviously improper as the images to be combined. Therefore, itis possible to reduce the load of the process.

In the above-mentioned example, the image selection unit 13 determineswhether all of the frames which are selected as the comparative imagecan be used as the images to be combined. However, the invention is notlimited thereto. For example, a predetermined number of frames may beset in advance and the process may proceed to Step S209 even when thereis a comparative image which has not been determined at the time thenumber of frames which are set as the images to be combined reaches thepredetermined number of frames. In particular, in this case, the processof determining whether the frame can be used as the images to becombined is preferentially performed for the comparative image which iscaptured at the time close to the capture time of the reference image,which makes it possible to improve the efficiency of the composite imagegeneration process.

The known technique may be used as the method of estimating the amountof movement of the object. For example, the movement of the object maybe estimated by comparing the brightness values of two consecutiveframes or by detecting the amount of movement of the imaging elementusing a sensor. In addition, the amount of movement of the object may bedirectly calculated or an element including another amount of movementwhich substitutes the amount of movement of the object may becalculated. For example, when the capture interval of the frame ispredetermined, a speed, that is, the speed of the object or the imagingelement may be calculated instead of the amount of movement.

Third Embodiment

Next, a third embodiment of the invention will be described withreference to the drawings. In this embodiment, the minimum total numberof necessary comparative images and images to be combined (hereinafter,referred to as the number of images required for composition) is set onthe basis of the amount of noise included in each frame.

FIG. 14 is a block diagram illustrating the structure of an imageprocessing device 30 according to this embodiment. In the example shownin FIG. 14, the image processing device 30 includes a compositioncontrol unit 11, a frame memory 12, and a noise amount estimation unit31. The composition control unit 11 and the frame memory 12 are the sameas those in the second embodiment.

The noise amount estimation unit 31 estimates the amount of noise in theframe. For example, the amount of noise is estimated on the basis of again value. The noise amount estimation unit 31 uses the gain valuewhich is determined by, for example, AE control. In general, as the gainvalue increases, the amount of noise increases. As the gain value isreduced, the amount of noise is reduced. For example, when the gainvalue is large, the amount of noise estimated by the noise amountestimation unit 31 is large. When the gain value is small, the amount ofnoise estimated by the noise amount estimation unit 31 is small. Anymethod may be used as long as it can estimate the amount of noise in theframe.

In general, as the number of frames to be composed (images to becombined) increases, the amount of noise to be removed increases.Therefore, in this embodiment, when the amount of noise estimated by thenoise amount estimation unit 31 is large, the image selection unit 13increases the number of images to be combined to a large value. When theamount of noise estimated by the noise amount estimation unit 31 issmall, the image selection unit 13 sets the number of images to becombined to a small value. Hereinafter, the sum of the number of imagesto be combined which are required in accordance with the amount of noiseand the number of reference images (one reference image) is referred toas the number of images K required for composition.

Next, a composite image generation process in this embodiment will bedescribed. FIG. 15 is a flowchart illustrating the composite imagegeneration process of the image processing device 30 according to thisembodiment.

The description of the same content as that in the first and secondembodiments will not be repeated.

(Step S301) The image processing device 10 stores frames (video signals)which are input one by one from the outside in the frame memory 12.

(Step S302) The noise amount estimation unit 31 estimates the amount ofnoise in each frame which the image processing device 10 stores in theframe memory 12. Then, the process proceeds to Step S303.

(Step S303) The image processing device 10 stores the amount of noisewhich is estimated by the noise amount estimation unit 31 in Step S302so as to be associated with each frame stored in Step S301, in the framememory 12. Then, the process proceeds to Step S304.

(Step S304) The composition control unit 11 determines whether acomposition instruction is received. When the composition control unit11 determines that a composition instruction is received, the processproceeds to Step S305. In the other cases, the process returns to StepS301.

(Step S305) The noise amount estimation unit 31 calculates the averagevalue of the amounts of noise of the frames stored in the frame memory12 on the basis of the amount of noise of each frame stored in the framememory 12. Then, the process proceeds to Step S306.

(Step S306) The image selection unit 13 calculates the number of imagesK required for composition on the basis of the average value of theamounts of noise calculated in Step S305. Then, the process proceeds toStep S307.

(Step S307) The image selection unit 13 selects, as the reference image,the frame which is captured at the time closest to the intermediate timebetween the time when the oldest frame is captured and the time when thenewest frame is captured among a plurality of frames which the imageprocessing device 10 store in the frame memory 12.

This process is the same as that in Step S103 of FIG. 7. Then, theprocess proceeds to Step S308.

(Step S308) The image selection unit 13 selects one comparative imagefrom the plurality of frames which the image processing device 10 storein the frame memory 12. Specifically, the comparative image ispreferentially selected from the frames which are captured at the timeclosest to the capture time of the reference image, among the framesother than the frame which is selected as the comparative image and theframe which has been subjected to the process in Step S309. This processis the same as that in Step S206 of FIG. 9. That is, the images to becombined is preferentially selected from the frames which are capturedat the time closest to the capture time of the reference image, whichwill be described below. Then, the process proceeds to Step S309.

(Step S309) The image selection unit 13 determines whether thecomparative image selected in Step S308 can be used as the images to becombined. Since the determination method is the same as that in StepS105 of FIG. 7, the description thereof will not be repeated in thisembodiment. The image selection unit 13 sets the comparative image whichis determined to be used as the images to be combined as the images tobe combined.

Then, the process proceeds to Step S310.

(Step S310) The image selection unit 13 determines whether the sum ofthe number of reference images and the number of set images to becombined is equal to the number of images K required for compositionwhich is calculated in Step S306. When the image selection unit 13determines that the sum is “equal to” the number of images K requiredfor composition, the process of determining whether the comparativeimage can be used as the images to be combined ends even though there isa frame which has not been selected as the comparative image, in theframe memory 12. Then, the process proceeds to Step S311. When the imageselection unit 13 determines that the sum is “not equal to the number ofimages K required for composition (the sum does not reach the number ofimages K required for composition)”, the process returns to Step S308and Steps S308 and S309 are repeatedly performed until the sum of thenumber of reference images and the number of set images to be combinedreaches the number of images K required for composition.

(Step S311) The composition processing unit 14 composes the ((K−1))images to be combined selected in Step S309 with the reference imageselected in Step S307 to generate a composite image. Then, the processends.

As described above, according to this embodiment, the image selectionunit 13 selects, as the reference image, the frame which is captured atthe time closest to the intermediate time between the time when theoldest frame is captured and the time when the newest frame is capturedamong a plurality of frames which the image processing device 10 storein the frame memory 12.

In addition, the noise amount estimation unit 31 estimates the amount ofnoise in each frame and calculates the average value of the amounts ofnoise.

The image selection unit 13 determines the number of frames (the numberof images K required for composition) used to generate a compositeimage, on the basis of the average value of the amounts of noisecalculated by the noise amount estimation unit 31. The compositionprocess is performed using only the number of images K required forcomposition. Therefore, since the composition processing unit 14generates a composite image using only the number of framescorresponding to the amount of noise, that is, only the minimum numberof necessary frames, it is possible to reduce the load of thecomposition process.

The images to be combined selected by the image selection unit 13 ispreferentially selected from the frames which are captured at the timeclose to the capture time of the reference image and have a relativelysmall amount of blur. Therefore, it is possible to improve theefficiency of the composite image generation process.

Fourth Embodiment

Next, a fourth embodiment of the invention will be described withreference to the drawings. In this embodiment, before a reference imageis selected, a frame section including a plurality of frames (the numberof frames which is less than the number of frames stored in a framememory) is set on the basis of the estimated amount of movement X of anobject. In other words, a reference image selection frame sectionincluding a plurality of frames which are consecutive in order ofcapture time is set on the basis of the estimated amount of movement Xof the object and the reference image is selected from a plurality offrames forming the reference image selection frame section.

In this embodiment, the number of frames forming the frame section isdescribed as the number of images K required for composition and thereference image selection frame section is described as a frame sectionin which the integrated value of the estimated amount of movement is theminimum.

FIG. 16 is a block diagram illustrating the structure of an imageprocessing device 40 according to this embodiment. In FIG. 16, the imageprocessing device 40 includes a composition control unit 11, a framememory 12, a movement amount estimation unit 21, and a noise amountestimation unit 31. The composition control unit 11, the frame memory12, and the movement amount estimation unit 21 are the same as those inthe second embodiment. The noise amount estimation unit 31 is the sameas the noise amount estimation unit 31 in the third embodiment.

FIG. 17 is a flowchart illustrating a composite image generation processof the image processing device 40 in this embodiment. The description ofthe same content as that described in the first to third embodimentswill not be repeated.

(Step S401) The image processing device 10 stores frames (video signals)which are input one by one from the outside in the frame memory 12.

(Steps S402 and S403) The movement amount estimation unit 21 estimatesthe amount of movement X of the object when each frame which is storedin frame memory 12 in Step S401 is captured. Then, the frame memory 12stores the estimated amount of movement X so as to be associated witheach frame. This process is the same as Steps S202 and S203 in FIG. 9.

(Step S404, S405) The noise amount estimation unit 31 estimates theamount of noise of each frame stored in the frame memory 12. Then, theframe memory 12 stores the estimated amount of noise so as to beassociated with each frame. This process is the same as Steps S302 andS303 in FIG. 15.

(Step S406) The composition control unit 11 determines whether acomposition instruction is received. When the composition control unit11 determines that a composition instruction is received, the processproceeds to Step S407. In the other cases, the process returns to StepS401.

(Step S407) The noise amount estimation unit 31 calculates the averagevalue of the amounts of noise of the frames stored in the frame memory12 on the basis of the amount of noise of each frame stored in the framememory 12. Then, the process proceeds to Step S408.

(Step S408) The image selection unit 13 calculates the minimum number ofnecessary frames required to generate a composite image, that is, thenumber of images K required for composition, which is the sum of thenumber of reference images and the number of images to be combined, onthe basis of the average value of the amounts of noise which iscalculated by the noise amount estimation unit 31 in Step S407. Then,the process proceeds to Step S409. This process is the same as Step S306in FIG. 15.

(Step S409) When selecting the reference image, first, the imageselection unit 13 specifies a frame section including K frames in whichthe amount of movement X is less than a prescribed amount of movementX_(a), which is a threshold value, from a plurality of frames stored inthe frame memory 12. The frame section is a plurality of frame groupshaving the oldest frame at the head. Then, the first frame in the framesection is shifted backward one by one to sequentially specify aplurality of frame sections. This process will be described withreference to FIG. 18.

FIG. 18 is a graph illustrating an example of the relationship between atime t and the amount of movement X of the imaging element 104 in eachframe stored in the frame memory 12. In the graph, a circle indicates aframe, a black circle indicates a frame in which the amount of movementX is less than the prescribed amount of movement X_(a), and a whitecircle indicates a frame in which the amount of movement X is equal toor more than the prescribed amount of movement X_(a).

First, the image selection unit 13 specifies a frame section T₀ in whichthe first frame is a frame F₀ which is captured at the earliest time t₀.The image selection unit 13 reads the estimated amount of movement X ofthe frame F₀, which is the first frame, from the frame memory 12 anddetermines whether the estimated amount of movement X is less than theprescribed amount of movement X_(a). When the estimated amount ofmovement X is less than the prescribed amount of movement X_(a), theimage selection unit 13 counts the frame F₀ as the frame in which theestimated amount of movement X is less than the prescribed amount ofmovement X_(a). When the determination process for the frame F₀ ends,the image selection unit 13 determines whether the estimated amount ofmovement X is less than the prescribed amount of movement X_(a) for thenext frame F₁ which is captured after the frame F₀. When the estimatedamount of movement X is less than the prescribed amount of movementX_(a), the image selection unit 13 counts the frame F₁ as the frame inwhich the estimated amount of movement X is less than the prescribedamount of movement X_(a). This process is repeatedly performed until thecounted number of frames reaches the number of images K required forcomposition. When the count result reaches the number of images Krequired for composition, a K-th frame is the last frame of the framesection T₀. In this way, the frame section T₀ is specified. FIG. 18shows an example in which K is 9 and the frame section T₀ is from theframe F₀ to a frame F₁₁.

When the frame section T₀ is specified, the image selection unit 13specifies the next frame section. The image selection unit 13 proceedsto a process of shifting the first frame to the next captured frame F₁of frame F₀ and specifying process frame section T₁ having the frame F₁as the first frame. Then, similarly to the operation of specifying theframe section T₀, the image selection unit 13 searches for a frame inwhich the amount of movement X is less than the prescribed amount ofmovement X_(a), counts the number of frames, sets the frame as the lastframe in the frame section at the time the count result reaches thenumber of images K required for composition, and specifies the framesection T₁. The image selection unit 13 repeatedly performs thespecifying process of this frame section until the frame F_(N) which iscaptured latest among the frames stored in the frame memory 12 becomesthe last frame of the frame section. Then, the image selection unit 13does not set, as the frame section, a frame section that does notinclude K frames in which the amount of movement X is less than theprescribed amount of movement X_(a) among the frame sections. Theabove-mentioned process is performed to specify the frame sections T₀,T₁, T₂, T₃ . . . .

(Step S410) Then, the image selection unit 13 specifies the referenceimage selection frame section in which the integrated value of theamount of movement X of the object is the minimum from the specifiedframe sections T₀, T₁, T₂, T₃ . . . . The image selection unit 13calculates the integrated value of the estimated amounts of movement foreach frame section, on the basis of each frame in which the amount ofmovement X is less than the prescribed amount of movement X_(a) in eachframe section. Then, the image selection unit 13 performs thecalculation process for all of the frame sections T₀, T₁, T₂, T₃ . . .which are obtained in Step S409. Then, the image selection unit 13specifies, as the reference image selection frame section, the framesection in which the calculated integrated value of the amounts ofmovement is the minimum.

FIG. 19 is a graph illustrating only the frame section T₀ having theframe F₀ as the first frame and the frame section T_(s) having the frameF_(s) as the first frame among a plurality of frame sections.

The amount of the integration of the amount of movement in the framesection T_(s) is less than the amount of integration of the amount ofmovement in the frame section T₀. In this case, the image selection unit13 specifies the frame section T_(s) as the reference image selectionframe section.

(Step S411) The image selection unit 13 selects the reference image froma plurality of frames included in the specified reference imageselection frame section. Specifically, first, the image selection unit13 excludes the frame which is captured earliest and the frame which iscaptured latest in the reference image selection frame section from thecandidates of the reference image. Then, the image selection unit 13selects the frame which is captured when the estimated amount ofmovement X of the object is the minimum from the remaining frames, asthe reference image.

This process will be described with reference to FIG. 20. In the graphshown in FIG. 20, a frame is indicated by a black circle in thereference image selection frame section T_(s) shown in FIG. 19. First,the image selection unit 13 excludes the first frame F_(s) and the lastframe F_(t) in the reference image selection frame section T_(s) fromthe comparative image candidates. Then, the image selection unit 13reads the estimated amount of movement relative to the remaining framesin the reference image selection frame section T_(s) from the framememory 12 and searches for the frame in which the estimated amount ofmovement is the minimum. Then, the image selection unit 13 selects, asthe reference image, a frame F_(m) which is captured at a time t_(m).

(Step S412) After the reference image selection process ends, the imageselection unit 13 proceeds to a comparative image selection process.First, the image selection unit 13 excludes the standard frame and theframe which has been subjected to the process in Step S413, which willbe described below, from a plurality of frames stored in the framememory 12. Then, the image selection unit 13 selects, from the remainingframes, the frame which is captured at the time closest to the capturetime of the reference image as a comparative image candidate. Then, whenthe amount of movement X is less than the prescribed amount of movementX_(a) in the frame which is selected as the comparative image candidate,the image selection unit 13 selects the selected frame as thecomparative image. When the amount of movement is equal to or more thanthe prescribed amount of movement X_(a) in the frame which is selectedas the comparative image candidate, the image selection unit 13determines that the frame is not the comparative image and selects thenext frame which is captured at the time close to the capture time ofthe reference image as the comparative image candidate. This process isthe same as Step S206 in FIG. 9.

The image selection unit 13 may end the comparative image selectionprocess when the frame selected as the comparative image candidate isdetermined not to be the comparative image, the next selectedcomparative image candidate is determined not to be the comparativeimage, and the comparative image candidate is determined not to be thecomparative image a prescribed consecutive number of times as a resultand proceed to Step S415. The prescribed number of times may bepredetermined or it may be arbitrarily set.

In this embodiment, the selected comparative image is preferentiallyselected from the frames in the previously obtained reference imageselection frame section. However, the comparative image may be selectedfrom the frames outside the reference image selection frame section. Theimage selection unit 13 selects all of the frames in the reference imageselection frame section as the comparative image and determines whetherthe comparative image is used as the images to be combined. As a result,when the number of frames which are set as the images to be combineddoes not reach the number of images K required for composition includingthe reference image, the image selection unit 13 selects, as thecomparative image, the frame which is outside the reference imageselection frame section, which is captured at the time closest to thecapture time of the reference image, and of which the amount of movementis less than the prescribed amount of movement X_(a). Then, the processproceeds to Step S413.

(Step S413) The image selection unit 13 determines whether thecomparative image selected in Step S412 can be used as the images to becombined. When it is determined that the selected comparative image canbe used as the images to be combined, the image selection unit 13 setsthe frame which is selected as the comparative image as the images to becombined. The determination method is the same as that in Step S207 inFIG. 9. Then, the process proceeds to Step S414.

When it is determined a prescribed consecutive number of times that theframe selected as the comparative image is not available as the imagesto be combined, the image selection unit 13 may end the process ofdetermining whether the comparative image can be used as the images tobe combined and proceed to Step S415. The prescribed number of times maybe predetermined or it may be arbitrarily set.

(Step S414) The image selection unit 13 determines whether the sum ofthe number of a reference image (single) and the number of set images tobe combined is equal to the number of images K required for compositioncalculated in Step S408. When the image selection unit 13 determinesthat the sum is “equal to” the number of images K required forcomposition, the process proceeds to Step S415. In the other cases, theprocess returns to Step S412 and Steps S412 and S413 are repeatedlyperformed until the sum of the number of reference images and the numberof set images to be combined reaches the number of images K required forcomposition.

(Step S415) The composition processing unit 14 composes ((K−1)) frameswhich are set as the images to be combined with the reference image togenerate a composite image.

As described above, according to this embodiment, the reference image isselected from a plurality of frames in which the amount of movement ofthe object is the minimum in the frame section (reference imageselection frame section) and the images to be combined is preferentiallyset from the frames which are captured at the time close to the capturetime of the reference image. Therefore, it is possible to preferentiallyset the images to be combined with a small amount of blur or a smallamount of movement relative to the reference image. As a result, it ispossible to generate a composite image using the frames which aresuitable for composition and thus to improve the efficiency of thecomposition processing.

Fifth Embodiment

Next, a fifth embodiment of the invention will be described withreference to the drawings. FIG. 21 is a block diagram illustrating thestructure of an endoscope apparatus according to this embodiment. In theexample shown in FIG. 21, an endoscope apparatus 101 includes anelongated insertion unit 102, a main body unit 103, and a liquid crystaldisplay (LCD) 130. In addition, a recording medium 132 can be attachedto the main body unit 103. For example, still images or moving imageswhich are captured by the endoscope apparatus 101 can be recorded on therecording medium 132.

The insertion unit 102 includes an objective lens 127, an imagingelement (charge coupled device (CCD)) 104, a light emitting diode (LED)106, and a wire fixing unit 119. The objective lens 127, the imagingelement 104, the LED 106, and the wire fixing unit 119 are arranged at atip 128 (endoscope tip) of the insertion unit 102.

The main body unit 103 includes an imaging element drive circuit 105, apre-amplifier 131, image processing means 107, an analog front end (AFE)109, system control means 110, a user interface 111, an LED drivecircuit 113, UD (up/down) curvature motor 120, an RL (right/left)curvature motor 121, curvature control means 122, image recording means115, and an LCD controller 117. In addition, the endoscope apparatus 101includes four wires 123 to 126 for curving the tip 128 of the insertionunit 102 in all directions.

The user interface 111 includes, for example, a switch or a joystick forcurving the tip of the endoscope, receives an instruction input from theuser, and outputs a signal corresponding to the received input to thesystem control means 110. Examples of the instruction from the userinclude an instruction for a zoom ratio, an instruction for thebrightness of the image to be captured, an instruction to turn on andoff the LED 106, an instruction to curve the tip 128 of the insertionunit 102, an instruction to record images on the recording medium 132,and an instruction to display images o the LCD 130.

The system control means 110 controls each unit of the endoscopeapparatus 101 such that a process based on the signal input from theuser interface 111 is performed. For example, when the user interface111 receives an input indicating the zoom ratio or an input indicatingthe brightness of the image to be captured, the system control means 110controls the image processing means 107 such that a process based on theinput received by the user interface 111 is performed. In addition, whenthe user interface 111 receives an input indicating the turn-on andturn-off of the LED 106, the system control means 110 controls the LEDdrive circuit 113 such that a process based on the input received by theuser interface 111 is performed. When the user interface 111 receives aninput indicating the curving of the tip 128 of the insertion unit 102,the system control means 110 controls the curvature control means 122such that a process based on the input received by the user interface111 is performed. When the user interface 111 receives an inputindicating the recording of images on the recording medium 132 or aninput indicating the display of images on the LCD 130, the systemcontrol means 110 controls the image recording means 115 such that aprocess based on the input received by the user interface 111 isperformed.

The wire fixing unit 119 fixes the four wires 123 to 126 for curving thetip 128 of the insertion unit 102 in all directions. Among the fourwires 123 to 126, two wires 123 and 124 for curving the tip 128 of theinsertion unit 102 in the up-down direction are connected to the UDcurvature motor 120. In addition, two wires 125 and 126 for curving thetip 128 of the insertion unit 102 in the left-right direction areconnected to the RL curvature motor 121.

The UD curvature motor 120 and the RL curvature motor 121 are connectedto the curvature control means 122. The curvature control means 122 isconnected to the system control means 110. The curvature control means122 controls the driving of the UD curvature motor 120 on the basis of acurvature instruction signal related to the up-down direction which isinput from the system control means 110 and controls the driving of theRL curvature motor 121 on the basis of a curvature instruction signalrelated to the left-right direction which is input from the systemcontrol means 110. The UD curvature motor 120 draws the two wires 123and 124 to curve the tip 128 of the insertion unit 102 in the up-downdirection under the control of the curvature control means 122. Inaddition, the RL curvature motor 121 draws the two wires 125 and 126 tocurve the tip 128 of the insertion unit 102 in the left-right directionunder the control of the curvature control means 122.

According to this structure, when the user tilts the joystick of theuser interface 111 in the up-down direction, the system control means110 transmits the curvature instruction signal related to the up-downdirection to the curvature control means 122. The curvature controlmeans 122 controls the driving of the UD curvature motor 120 on thebasis of the received curvature instruction signal and draws the wires123 and 124 connected to the UD curvature motor 120. In this way, theendoscope apparatus 101 can curve the tip 128 of the insertion unit 102in the up-down direction. The tip is curved in the left-right directionby the same method as described above. When the user tilts the joystickin the left-right direction, the system control means 110 transmits thecurvature instruction signal related to the left-right direction to thecurvature control means 122. The curvature control means 122 controlsthe driving of the RL curvature motor 121 on the basis of the receivedcurvature instruction signal and draws the wires connected to the RLcurvature motor 121. In this way, the endoscope apparatus 101 can curvethe tip 128 of the insertion unit 102 in the left-right direction.

The LED 106 is connected to the LED drive circuit 113 through a cable.The LED drive circuit 113 is connected to the system control means 110.The LED drive circuit 113 controls the turn-on and turn-off of the LED106 on the basis of an LED turn-on signal input from the system controlmeans 110. The LED 106 is turned on and off under the control of the LEDdrive circuit 113.

The objective lens 127 forms an object image illustrated by the LED 106on a light receiving surface of the imaging element 104. The imagingelement 104 is connected to the imaging element drive circuit 105 andthe pre-amplifier 131 through a combined coaxial cable. The imagingelement drive circuit 105 receives a timing signal for driving theimaging element 104 from a timing generator which is provided in theimage processing means 107. Then, the imaging element drive circuit 105performs a drive process corresponding to the length of a transmissionpath (the length of the combined coaxial cable) to the imaging element104 for the received timing signal and transmits the processed timingsignal as an imaging element driving signal to the imaging element 104.

The imaging element 104 converts the light focused on the lightreceiving surface into an electric signal on the basis of the timing ofthe transmitted imaging element driving signal and outputs the convertedelectric signal as an imaging element output signal. The imaging elementoutput signal output from the imaging element 104 is input to thepre-amplifier 131 through the combined coaxial cable. The pre-amplifier131 amplifies the imaging element output signal in order to compensatefor a signal level which is attenuated by transmission through thecombined coaxial cable. The imaging element output signal amplified bythe pre-amplifier 131 is input to the AFE 109. The AFE 109 performs acorrelated double sampling (CDS) process, an auto gain control (AGC)process, or an analog/digital (AD) conversion process for the inputimaging element output signal. The imaging element output signal (frame)processed by the AFE 109 is input to the image processing means 107.

The image processing means 107 includes at least one of the imageprocessing device 10 according to the first embodiment, the imageprocessing device 20 according to the second embodiment, the imageprocessing device 30 according to the third embodiment, and the imageprocessing device 40 according to the fourth embodiment. The imageprocessing means 107 generates a composite image using the input imagingelement output signal (frame) and removes noise, under the control ofthe system control means 110. The image processing means 107 accordingto this embodiment generates the composite image using any one of aplurality of generation methods according to the first to fourthembodiments, according to the image processing devices 10, 20, 30, and40 provided therein.

The image processing means 107 may perform various kinds of camerasignal processing, such as white balance correction, gamma correction,contour correction, electronic zoom processing, color correction,contrast correction, and AE control. In addition, the image processingmeans 107 outputs the generated composite image as image data to theimage recording means 115.

The image recording means 115 outputs the input image data to the LCDcontroller 117. In addition, the image recording means 115 performs afreeze process of displaying a still image on the LCD 130. The imagerecording means 115 compresses the input image data with an encoder andrecords the image data as still image data or moving image data on therecording medium 132. This image recording operation is performed on thebasis of the input from the user interface 111 when the system controlmeans 110 transmits a recording signal to the image recording means 115and the image recording means 115 receives the recording signal.

The endoscope apparatus 101 captures a still image when image recordingis performed after the image recording means 115 performs the freezeprocess and captures a moving image when image recording is performedwithout the freeze process. The image recording means 115 performs animage reproduction operation of reading still image data or moving imagedata from the recording medium 132, decompressing the data, andoutputting the decompressed data to the LCD controller 117. This imagereproduction operation is performed when the system control means 110receives an image display signal from the user interface 111.

The LCD controller 117 performs image processing (for example, gammacorrection, scaling, and RGB conversion) which is most suitable for theconnected LCD 130 for various kinds of image data input from the imagerecording means 115 and outputs the processed image data to the LCD 130.The LCD 130 displays an image on the basis of the input image data.

As described above, the image processing means 107 of the endoscopeapparatus 101 includes at least one of the image processing device 10according to the first embodiment, the image processing device 20according to the second embodiment, the image processing device 30according to the third embodiment, and the image processing device 40according to the fourth embodiment. Therefore, the image processingmeans 107 can generate a composite image using the frame which is moresuitable for composition among the frames captured by the imagingelement 104 of the endoscope apparatus 101. In addition, the imageprocessing means 107 can obtain the same effect as the image processingdevices 10, 20, 30, and 40, according to the image processing devices10, 20, 30, and 40 provided therein.

A plurality of embodiments of the invention have been described abovewith reference to the drawings. However, the detailed structure is notlimited to the plurality of embodiments and the invention also includes,for example, the structure which is designed without departing from thescope and spirit of the invention.

All or some of the functions of each unit provided in the imageprocessing devices 10, 20, 30, and 40 according to the above-describedembodiments may be implemented by recording a program for implementingthe functions on a computer-readable recording medium and allowing acomputer system to reading the program recorded on the recording mediumand to execute the program. The term “computer system” includes an OSand hardware, such as peripheral devices.

Typically, examples of the “computer-readable recording medium” includeportable media, such as a flexible disk, a magneto-optical disk, a ROM,and a CD-ROM, and a recording unit, such as a hard disk provided in thecomputer system. However, the computer-readable recording medium is notnecessarily limited thereto.

In addition, the following means may be used instead of the“computer-readable recording medium”. For example, the computer-readablerecording medium may be a medium that dynamically stores a program for ashort period of time, such as a communication line used when a programis transmitted through a network such as the Internet or a communicationline such as a telephone line. In addition, in this case, thecomputer-readable recording medium may be a medium that stores a programfor a predetermined period of time, such as a volatile memory providedin a computer system serving as a server or client. Furthermore, theprogram may be executed to implement some of the above-mentionedfunctions. Further, the above-mentioned functions may be implemented bycombinations of all programs recorded on the computer system.

While preferred embodiments of the invention have been described andillustrated above, it should be understood that these are exemplary ofthe invention and are not to be considered as limiting. Additions,omissions, substitutions, and other modifications can be made withoutdeparting from the spirit or scope of the present invention.Accordingly, the invention is not to be considered as being limited bythe foregoing description, and is only limited by the scope of theappended claims.

What is claimed is:
 1. An image processing device comprising: a memoryconfigured to store a plurality of captured images; and a processorcomprising hardware, wherein the processor is configured to implement:an image selection unit configured to: select a reference image fromamong the plurality of captured images stored in the memory; and selectimages to be combined with the reference image from: a first group ofimages among the plurality of captured images, the first group of imagesbeing captured earlier than the reference image; and a second group ofimages among the plurality of captured images, the second group ofimages being captured later than the reference image; and a compositionprocessing unit configured to compose the images to be combined selectedby the image selection unit with the reference image selected by theimage selection unit to generate a composite image.
 2. An endoscopeapparatus comprising: an insertion unit; an imaging unit disposed at atip of the insertion unit; an image processing device, wherein the imageprocessing device includes: a memory configured to store a plurality ofcaptured images which are captured by the imaging portion, a processorcomprising hardware, wherein the processor is configured to implement:an image selection unit configured to: select a reference image fromamong the plurality of captured images stored in the memory; and selectimages to be combined with the reference image from:  a first group ofimages among the plurality of captured images, the first group of imagesbeing captured earlier than the reference image; and  a second group ofimages among the plurality of captured images, the second group ofimages being captured later than the reference image; and a compositionprocessing unit configured to compose the images to be combined selectedby the image selection unit with the reference image selected by theimage selection unit to generate a composite image, and an image displayunit configured to display the composite image.
 3. An image processingmethod comprising: selecting a reference image from among the pluralityof captured images stored in the memory; selecting images to be combinedwith the reference image from: a first group of images among theplurality of captured images, the first group of images being capturedearlier than the reference image; and a second group of images among theplurality of captured images, the second group of images being capturedlater than the reference image; and among the plurality of images; andcomposing the images to be combined with the selected reference image togenerate a composite image.
 4. A computer-readable device storing aprogram causing a computer to perform steps of: selecting a referenceimage from among the plurality of captured images stored in the memory;selecting images to be combined with the reference image from: a firstgroup of images among the plurality of captured images, the first groupof images being captured earlier than the reference image; and a secondgroup of images among the plurality of captured images, the second groupof images being captured later than the reference image; and among theplurality of images; and composing the images to be combined with theselected reference image to generate a composite image.